Confidence based selection for survey sampling

ABSTRACT

A system and method for confidence-based selection of items for use in conducting a computer-implemented survey. The survey presents information about a selected plurality of items to a survey participant, to elicit survey feedback information. Information regarding the plurality of items is stored, the stored information including display information about each of the plurality of items for presentation to a survey participant. A subset of items for presentation to a survey participant is selected in accordance with a predetermined selection algorithm. Information corresponding to the selected subset of items is displayed to the survey participant via a survey user interface. Rating information is input by the survey participant via the survey user interface indicating the survey participant&#39;s preferences as to items in the presented subset of items. The rating information is utilized in various manners to affect the selection algorithm for a subsequent survey.

FIELD OF THE INVENTION

This invention relates to a computer implementation of algorithms andmethods for collecting and ranking ideas, people, or any other items.This invention further relates to use of a survey-like mechanism toobtain a prioritized or ranked list of items. The mechanism may beimplemented using a network, such as the Internet.

BACKGROUND

Many entities, including those that conduct business over the Internet,find it beneficial to conduct surveys to determine what products shouldbe offered, what services should be made available, what content shouldbe present on a website, and so forth. Such surveys may be presented toa user after a transaction, such as the purchase of a product.Alternatively, the survey may be presented to a user independent of anyparticular transaction. Existing survey systems use a variety ofdelivery mechanisms including email invitations, banner adds, popupwindows, and links on websites. The layout and presentation of thesurvey questions may be customized by the author of the survey to theextent allowable by the electronic survey system.

Typical survey systems allow the survey participant to respond viaemail, website, web application, web applet, or a client specificallydesigned to accept, transmit, or store the responses. The responses maytake on a variety of forms including, but not limited to, single choice,multiple choice, rating scale, and text responses. Electronic surveysystems typically gather the responses in a database or other electronicstorage mechanism. The response data often becomes the subject matterfor various reports, graphs, charts, and analysis. The reportinganalysis tools are often separate from the actual survey systemsthemselves.

Various survey systems are presently available that allow electronicsurveys to be authored and transmitted over the Internet. The existingsystems use a variety of participation models. Some of the systems usean invitation-based system, in which an invitation is sent to theparticipant via email or other form of electronic message. The emailinvitations may or may not contain information to identify the personinvited to the survey. Some systems use an open model that allows anyvisitor to a website to follow a link and respond to an electronicsurvey without ever identifying himself. There is also aself-registration model, where the participants identify themselvesduring a registration process before taking the survey.

There are several challenges associated with the use of surveys togather information. It has been established that response rates tosurveys typically decline as the amount of time required to respondincreases. If the survey is too long or time-consuming, the user may notcomplete the survey. At the same time, the number of respondents isoften a critical factor in the accuracy of a survey. The survey may notbe effective if too few questions are asked of the user. Further, thequestions and answer choices are predetermined by the author prior tothe survey being made available for participants to respond. Thus,additional ideas of the user may not be captured.

Various efforts have been made to address these challenges. For example,to minimize the number of questions asked, some presently availableelectronic survey systems enable the author of the survey to specify thepresentation order and skip certain survey questions based on theanswers to previous questions. But even in such systems, the author ofthe survey writes all of the questions and answer choices.

Another way to reduce the length of the survey is to show theparticipant only a subset of the entire list of possible questions.There is one known company who offers the ability to show a participanta subset of the entire list. Informative, Inc. (Brisbane, Calif.) offersa product that allows a participant to select a subset of items from alarger list (that is a subset of all available items), and then arrangethe subset into order according to the participant's preference.

Such systems are based on the premise that items that are receiving highrankings from respondents should be presented more often than items thatare receiving low rankings. These systems consider a data collectioneffort to be either mature or immature, depending on the number ofresponses. This maturity status applies to the entire set of items. Whenthe data collection effort is in the immature state, the items arepresented to the respondents at random. When a sufficient number ofresponses is collected and the data collection is considered mature, theselection process shifts to selecting items with higher rankings topresent to subsequent respondents. While this may be appropriate in someinstances, it is limited in applicability and does not consider someother important factors that can be used to select a sample.

Some electronic survey systems have attempted to gather new items from apopulation of participants by enabling the participant to enter a textanswer. However, to enable rapid processing of the survey results, suchquestions are typically limited in number, which places an artificiallimitation on the number of items any one participant can submit.Furthermore, the items that are input are often deposited in a databasewith little information about the importance of the items.

Thus, there remains a need for a process for ranking items, people, orany other items in a manner that encourages participation and achieveshigh response rates. There further remains a need for a system that isable to collect items from the user and incorporate such items such thatthe new items are available for ranking by other participants.

SUMMARY OF THE INVENTION

The present invention generally relates to a computer based system forestimating the preference of a list of items as perceived by apopulation of participants, even though each participant rates only asubset of the entire list of items available for presentation. Sincecomplete data will not likely be available, and the participants who dorespond will most likely not completely agree on which items are mostimportant, the present invention uses statistical computations to usethe data that is available to most accurately estimate the order ofpreference that best fits the entire population. The amount of data andthe quality of the data actually collected determines how accurate theestimated order actually reflects the order of preference for thepopulation.

The system of the present invention increases the number of participantresponses by decreasing the amount of time required for each respondentto express his preferences. To do so, only a subset of the complete listof items is presented to each participant, so that each participant ismore likely participate and provide data. By increasing participation,the data is less likely to be biased toward the preferences of a few ofthe participants.

The system of the present invention is focused in particular onselecting the subset of items to be presented to each person of thepopulation. The items are selected in a manner that improves theconfidence in the estimated order of preference of the entire set ofitems.

According to one aspect, the present invention is a method forconducting a computer-implemented survey relating to a plurality ofitems from a plurality of survey participants. The invention of thisaspect comprises a number of steps that include storing informationregarding the plurality of items, the stored information includingdisplay information about each of the plurality of items forpresentation to a survey participant and presentation number informationcorresponding to the number of times a particular item has beenpreviously shown to survey participants, selecting a subset of items forpresenting to a survey participant in accordance with a predeterminedselection algorithm that utilizes the presentation number information toinfluence the selection of items for the subset, presenting the displayinformation corresponding to the selected subset of items is presentedto the survey participant via a survey user interface, and receivingrating information input by the survey participant via the survey userinterface indicating the survey participant's preferences as to items inthe subset of items presented.

In accordance with this first aspect, the method may further include thestep of utilizing the rating information input by the survey participantto affect the probability of selection of the items in the selectedsubset in a subsequent selection of a subset of items for presentationto a subsequent survey participant using the predetermined selectionalgorithm.

In further accordance with this first aspect, the predeterminedselection algorithm utilizes an adjustment factor to cause items to beselected more or less often as a function of the rating informationobtained by previous participants of a survey. Further still, thepredetermined selection algorithm utilizes, at least in part, a randomnumber selection of items in the plurality of items.

The plurality of items may include a predetermined duplication of itemsin a set of the plurality of items, with the number of duplications ofparticular items influenced by the rating information.

In further accordance with this first aspect, the predeterminedselection algorithm is self-adjusting based on previous responsesreceived during a previous survey of the plurality of items. Theself-adjusting may be based on modifying the probability of selecting anitem from the plurality of items, the modifying in turn based collectionof new items input by survey participants and rating of new items incomparison of previously included items.

In further accordance with this first aspect, the predeterminedselection algorithm is operative initially to randomly select items forthe subset of items, and thereafter operative to select items based onutilization of the rating information. The probability of selection of agiven item may be continuously adjusted as items are rated by surveyparticipants.

In accordance with a second aspect of the invention, the presentinvention is another method for conducting a computer-implemented surveyrelating to a plurality of items from a plurality of surveyparticipants. The invention of this aspect comprises a number of stepsthat include storing information regarding the plurality of items in amemory, the stored information including display information about eachof the plurality of items for presentation to a survey participant andfrequency information corresponding to the number of times a particularitem has been previously shown to survey participants. The methodfurther includes the step of selecting a subset of items for presentingto a survey participant in accordance with a predetermined selectionalgorithm that utilizes the frequency information. The method furtherincludes the step of presenting the display information corresponding tothe selected subset of items to the survey participant via a survey userinterface. The method further includes the step of receiving ratinginformation input by the survey participant via the survey userinterface indicating the survey participant's preferences as to items inthe subset of items presented. The method further includes the step ofutilizing the rating information input by the survey participant toaffect the probability of selection of the items in the selected subsetin a subsequent selection of a subset of items for presentation to asubsequent survey participant using the predetermined selectionalgorithm.

In accordance with this second aspect, the rating information isselected from the group comprising: ranking of items relative to eachother, ranking of the items on a scale, grading the items, ordering ofthe items, allocating points among items, scaling the items, choosing anitem over other items, categorizing items, and other equivalent methodsof indicating a preference of one item over another.

In further accordance with this second aspect, the step of selecting asubset of items in accordance with the predetermined selection algorithmcomprises selecting based on a ranking of items using rating informationfrom previous participants, such that the probability of selection ofparticular items for presenting in a subsequent survey is influenced bythe rating information. In this manner, certain items that have beenrated lower than other items are more likely to be selected for a surveyso as to increase the number of presentations of such items.

In further accordance with this second aspect, the method comprises thestep of providing the rating information for each item as an output ofthe method indicative of survey results.

In the foregoing and most aspects of the invention, the memory is arandom access memory array. Information regarding the plurality of itemsis stored in an ordered array, for example in the memory, and selectedaccording to a probability index. The information about each item in theordered array is stored in a data field in the ordered array.

In further accordance with the second aspect, the subset of itemsselected for presentation to the survey participant is an initialsubset, and the subsequent selection of a set of items for presentationcomprises a selection from the plurality of items that may include oneor more of the items from the initial subset.

In further accordance with the second aspect, the information regardingthe plurality of items includes a unique identifier for each item foruse as a primary key to access the item in the memory.

In further accordance with the second aspect, the method furthercomprises the step of storing a users item table for storing informationprovided by a survey participant relating to an additional item forinclusion in the plurality of items.

In accordance with various aspects of the invention, not limited to thefirst or second, the information regarding the plurality of itemsincludes status information about each item. The status information isindicative whether an item has been previously shown to a surveyparticipant or not. The predetermined selection algorithm also utilizesthe status information in conjunction with the frequency information.The frequency information may be stored in a times-shown field for eachitem in the array.

In further accordance with the second aspect, the predeterminedselection algorithm utilizes a selection score in selecting items forpresentation. The selection score is based upon a confidence score. Theselection score is further based on an adjusted mean rating determinedfrom the rating information. The selection score is further based on arank influence factor. The rank influence factor is an arbitrary numberused to adjust the probability of an item being selected based onranking information. The ranking information comprises informationcorresponding to the actual ranking of an item in relation to otheritems in the plurality of items.

In further accordance with the second aspect, the selection algorithmselects an item from the plurality of items based on a computation of aprobability index. The probability index is determined based on anormalized selection score. The normalized selection score is utilizedto determine a probability of selection for each item in the pluralityof items, the probability of selection of each item is used to determinehow many times an item is represented in the plurality of items forselection.

In further accordance with various aspects of the invention, not limitedto the first or second aspect, the plurality of items are represented ina computer system as a pool of selectable items stored in an array ofitems, with each item in the pool having a high index number and a lowindex number, with the index numbers representing how many times an itemis represented in the pool of selectable items, and wherein the step ofselecting a subset of items comprises selecting from the pool of itemsbased on a random number used to index into the array of index numbers.The subset of items for presenting to a survey participant is selectedby repeating the step of selecting utilizing the random number, until apredetermined number of items corresponding to the size of the selectedsubset of items has been chosen for presentation.

In further accordance with various aspects of the invention, not limitedto the first or second, a selected subset of items comprises a uniquesample of items in the plurality of items of a predetermined sample sizethat meets predetermined selection criteria according to statusinformation associated with the item. The predetermined sample sizecomprises the maximum number of items presented to a survey participantin the survey. The status information comprises information indicativeof a condition associated with an item. The status information may beselected according to various criteria, for example, whether an item isscheduled, whether an item has been approved, whether an item isimplemented, whether an item is active, whether an item is in or underreview, whether an item has been submitted, whether an item has beendeclined, or other equivalent information indicative of a condition ofan item.

In further accordance with various aspects of the invention, not limitedto the first or second, the selected subset of items is selected forpresentation based at least in part on an indication of interest of aparticipant. The indication of interest of a participant is obtained byinput of interest information by a survey participant in response to aquery prior to selection of the subset. The indication of interest of aparticipant is obtained by examining items previously submitted by thesurvey participant, and by selecting other items from the plurality ofitems based on the topical similarity of other items in the plurality ofitems. The indication of interest of a participant may be obtained byexecuting a query of keywords relating to items submitted by the surveyparticipant.

In further accordance with various aspects of the invention, not limitedto the first or second, the selected subset of items for presentation isa first selected subset of a predetermined small number of items, where“small” is relative but determined based on a number that is deemed by asurvey manager to be acceptable for purposes of a particular survey, andfurther comprising the step of selecting additional items forpresentation to a survey participant. The step of selecting additionalitems for presentation to a survey participant is based on informationprovided by a survey participant indicating a desire to view and ratemore items. The survey participant is provided with a display offeringan opportunity to request an additional sample of items for rating, andwherein the information provided by the survey participant indicating adesire to view and rate more items is input by the survey userinterface. The opportunity to request an additional sample of items forrating is typically providing during a survey session.

According to a third aspect, the present invention is a method forconducting a computer-implemented survey of a plurality of items. Theinvention of this aspect comprises a number of steps that includearranging the plurality of items in a memory in ordered array, providinga unique identifier for each item in the array; providing informationabout each item in the array for presentation to a survey participant,providing a status information field for each item in the array,providing a times-shown field for each item in the array, selecting asubset of items for presenting to a survey participant in accordancewith a predetermined selection algorithm that utilizes the informationin the times-shown field of the items, presenting informationcorresponding to the selected subset of items to the survey participantvia a survey user interface, receiving rating information input by thesurvey participant via the survey user interface indicating the surveyparticipant's preferences as to the subset of items presented, andutilizing the rating information input by the survey participant toaffect the probability of selection of the items in the selected subsetof times for a subsequent selection of a set of items for presentationto a subsequent survey participant using the predetermined selectionalgorithm.

According to a fourth aspect, the present invention is a method forconducting a computer-implemented survey relating to a plurality ofitems from a plurality of survey participants. The invention of thisaspect comprises a number of steps that include receiving ratinginformation input by a particular subset of a plurality of surveyparticipants via a survey user interface indicating the surveyparticipants' preferences as to items in a presented subset of items ofthe plurality of items, computing the mean of the rating information,computing the standard error of the mean of the rating information,determining a confidence score utilizing the standard error of therating information, and utilizing the confidence score to select adifferent subset of the plurality of items for presentation to adifferent subset of the plurality of survey participants. In thismanner, items that might benefit from additional ratings by additionalparticipants are selected for presentation in a subsequent survey.

According to a fifth aspect, the present invention is a method fordynamically selecting a subset of items from a plurality of items forpresentation to a survey participant in a computer-implemented surveyrelating to the plurality of items. The invention of this aspectcomprises a number of steps that include storing rating informationinput by a plurality of prior survey participants indicating such priorsurvey participants' preferences as to items in one or more subsets ofitems presented to such survey participants in a prior survey,determining the number of participants that have previously ratedparticular items in the plurality of items, determining a measure ofagreement by the determined number of participants on the stored ratinginformation of the particular previously rated items in the plurality ofitems, utilizing the measure of agreement on the ratings of suchpreviously rated particular items to adjust the probability of selectionof such previously rated particular items for a subsequent selection,and selecting a subset of items for presentation to the surveyparticipant as a function of the adjusted probability of selection.

In accordance with this and various other aspects of the invention, themeasure of agreement comprises the standard deviation of the meanratings provided by the prior survey participants, or alternativelycomprises the average of the mean ratings provided by the prior surveyparticipants.

In accordance with this and various other aspects of the invention, thestep of continuously changing the items in the selected subset of itemsbased on a determined measure of agreement on the stored ratinginformation on previously rated items.

According to a sixth aspect, the present invention is a method fordynamically selecting a subset of items from a plurality of items forpresentation to a survey participant in a computer-implemented surveyrelating to the plurality of items. The invention of this aspectcomprises a number of steps that include storing rating informationinput by a plurality of prior survey participants indicating such priorsurvey participants' preferences as to items in one or more subsets ofitems presented to such survey participants in a prior survey,determining the number of participants that have previously ratedparticular items in the plurality of items, determining a measure ofagreement by the determined number of participants on the stored ratinginformation of the particular previously rated items in the plurality ofitems, continuously adjusting the probability of selection of suchpreviously rated particular items for a subsequent selection based onthe determined measure of agreement on the stored rating information onpreviously rated items, and selecting a subset of items for presentationto the survey participant as a function of the adjusted probability ofselection.

According to this and various other aspects of the invention, themeasure of agreement comprises the standard deviation of the meanratings provided by the prior survey participants, or the average of themean ratings provided by the prior survey participants.

This sixth and other aspects of the invention may also include the stepof continuously changing the items in the selected subset of items basedon the determined measure of agreement on the stored rating informationon previously rated items.

According to a seventh aspect, the present invention is a method forconducting a computer-implemented survey relating to a plurality ofitems from a plurality of survey participants in a manner that surveyparticipants can contribute new items. The invention of this aspectincludes a number of steps including storing information regarding theplurality of items in a memory, the stored information including displayinformation about each of the plurality of items for presentation to asurvey participant, selecting a subset of items for presenting to asurvey participant in accordance with a predetermined selectionalgorithm, presenting the display information corresponding to theselected subset of items to the survey participant via a survey userinterface, receiving rating information input by the survey participantvia the survey user interface indicating the survey participant'spreferences as to items in the subset of items presented, receivinginformation input by the survey participant corresponding to anadditional item for inclusion in the plurality of items, and selecting asecond subset of items from the plurality of items that now includes theadditional item for presenting to a subsequent survey participant inaccordance with the predetermined selection algorithm.

According to this seventh and various other aspects of the invention,the method may further include the step of utilizing the ratinginformation input by the survey participant to affect the probability ofselection of the items in a subsequent selection of a subset of itemsfor presentation to a subsequent survey participant. The informationinput by the survey participant is provided via an additional itemsubmission user interface.

According to an eighth aspect, the present invention is a method forconducting a computer-implemented survey relating to a plurality ofitems from a plurality of survey participants, with an adjustment factorapplied to the probability of selection. The invention of this aspectcomprises a number of steps including storing information regarding theplurality of items in a memory, the stored information including displayinformation about each of the plurality of items for presentation to asurvey participant, conducting a selection operation involving selectinga subset of items for presenting to a survey participant in accordancewith a function that utilizes a probability of selection, presenting thedisplay information corresponding to the selected subset of items to thesurvey participant via a survey user interface, receiving ratinginformation input by the survey participant via the survey userinterface indicating the survey participant's preferences as to items inthe subset of items presented, determining an adjustment factor for theprobability of selection of items in the subset of items as a functionof the number of times that the items have already been selected andpresented to previous survey participants; and applying the adjustmentfactor to the probability of selection for a subsequent selectionoperation.

According to a ninth aspect, the present invention is a method forconducting a computer-implemented survey relating to a plurality ofitems from a plurality of survey participants, wherein the selection ofitems is influenced by an adjustment factor derived from previousratings. The invention of this aspect comprises a number of steps thatinclude storing information regarding the plurality of items, the storedinformation including display information about each of the plurality ofitems for presentation to a survey participant, conducting a selectionoperation involving selecting a subset of items for presenting to asurvey participant in accordance with a function that utilizes aprobability of selection, presenting the display informationcorresponding to the selected subset of items to the survey participantvia a survey user interface, receiving rating information input by thesurvey participant via the survey user interface indicating the surveyparticipant's preferences as to items in the subset of items presented,determining an adjustment factor for the probability of selection ofitems in the subset of items as a function of the rating information onitems that have already been selected and presented to previous surveyparticipants; and applying the adjustment factor to the probability ofselection for a subsequent selection operation.

According to a tenth aspect, the present invention is a method forconducting a computer-implemented survey relating to a plurality ofitems from a plurality of survey participants, having a transitionalselection process. The invention of this aspect comprises a number ofsteps including storing information regarding the plurality of items,the stored information including display information about each of theplurality of items for presentation to a survey participant, conductinga selection operation involving selecting a subset of items forpresenting to a survey participant in accordance with a function thatutilizes a probability of selection, the selection operation initiallyoperative to select a subset of items on random basis, presenting thedisplay information corresponding to the selected subset of items to thesurvey participant via a survey user interface, receiving ratinginformation input by the survey participant via the survey userinterface indicating the survey participant's preferences as to items inthe subset of items presented; determining an adjustment factor for theprobability of selection of items in the subset of items as a functionof the rating information on items that have already been selected andpresented to previous survey participants and on the number of surveyparticipants that have rated particular items, and applying theadjustment factor to the probability of selection for a subsequentselection operation. In this manner, the probability of each item beingselected is continuously adjusted to be less random and more biasedtoward selection of unrated items and/or infrequently viewed items asitems are rated by survey participants and rating information onparticular items is collected.

According to an eleventh aspect, the present invention is a method forconducting a computer-implemented survey relating to a plurality ofitems from a plurality of survey participants, involving definedcriteria or predefined attributes for selection of items. The inventionof this aspect includes a number of steps including storing informationregarding the plurality of items in a memory, the stored informationincluding display information about each of the plurality of items forpresentation to a survey participant and predefined attributeinformation relating to predetermined attributes of each of theplurality of items, conducting a selection operation involving selectinga subset of items for presenting to a survey participant in accordancewith a function that utilizes the predefined attribute information,presenting the display information corresponding to the selected subsetof items to the survey participant via a survey user interface, andreceiving rating information input by the survey participant via thesurvey user interface indicating the survey participant's preferences asto items in the subset of items presented.

In accordance with this aspect of the invention, the selection operationis further a function of probability of selection of the plurality ofitems as well as the attribute information. The method may furtherinclude steps of determining an adjustment factor for the probability ofselection of items in the subset of items as a function of the ratinginformation on items that have already been selected and presented toprevious survey participants and on the number of survey participantsthat have rated particular items, and applying the adjustment factor tothe probability of selection for a subsequent selection operation. Inthis manner, the probability of each item being selected is continuouslyadjusted to be less random and more biased toward selection of unrateditems and/or infrequently viewed items as items are rated by surveyparticipants and rating information on particular items is collected.The attribute information may comprises one or more of the following: acategory, originator, priority, purpose, or other types of criteria orattributes.

According to a twelfth aspect, the present invention is a method forconducting a computer-implemented survey relating to a plurality ofitems from a plurality of survey participants, wherein an indication ofparticipant willingness is utilized. The invention of this aspectcomprises a number of steps that include (a) storing informationregarding the plurality of items in a memory, the stored informationincluding display information about each of the plurality of items forpresentation to a survey participant; (b) conducting a selectionoperation involving selecting a subset of items for presenting to asurvey participant in accordance with a predetermined function; (c)presenting the display information corresponding to the selected subsetof items to the survey participant via a survey user interface; (d)receiving rating information input by the survey participant via thesurvey user interface indicating the survey participant's preferences asto items in the subset of items presented; (e) receiving an indicationinput by a survey participant of willingness to view and rate additionalitems; and (f) in response to receipt of the indication input by asurvey participant of willingness to view and rate additional items,conducting a subsequent selection operation (b) and repeating the steps(c) through (f).

According to a thirteenth aspect, the present invention is a method forconducting a computer-implemented survey relating to a plurality ofitems from a plurality of survey participants, wherein a determinationthat more rating data is needed is made and utilized. The invention ofthis aspect comprises a number of steps that include storing informationregarding the plurality of items in a memory, the stored informationincluding display information about each of the plurality of items forpresentation to a survey participant, conducting a selection operationcomprising selecting a subset of items for presenting to a surveyparticipant in accordance with a predetermined selection algorithm,presenting the display information corresponding to the selected subsetof items to the survey participant via a survey user interface,receiving rating information input by the survey participant via thesurvey user interface indicating the survey participant's preferences asto items in the subset of items presented, determining that particularitem in the plurality of items should be presented more frequently so asto obtain additional rating data; and adjusting a parameter of thepredetermined selection algorithm so as to increase the likelihood thatthe particular item will be selected during a subsequent selectionoperation for a subsequent survey.

According to this and various other aspects of the invention, the stepof adjusting a parameter of the selection algorithm comprise computing aconfidence score among two or more items in the plurality of items,comparing the confidence scores, and using the results of the comparisonto adjust the probability of selection of a particular item for whichadditional data is needed. The parameter of the selection algorithm maybe adjusted as a function of the number of survey participants that havepreviously rated the particular item. The parameter of the selectionalgorithm may be adjusted as a function of the ratings of the particularitem by survey participants that have previously rated the particularitem.

According to a fourteenth aspect, the present invention is a method forconducting a computer-implemented survey relating to a plurality ofitems from a plurality of survey participants, wherein a determinationthat more data is needed for newer items is made and utilized. Theinvention of this aspect comprises a number of steps that includestoring information regarding the plurality of items in a memory, thestored information including display information about each of theplurality of items for presentation to a survey participant, conductinga selection operation comprising selecting a subset of items forpresenting to a survey participant in accordance with a predeterminedselection algorithm, presenting the display information corresponding tothe selected subset of items to the survey participant via a survey userinterface, receiving rating information input by the survey participantvia the survey user interface indicating the survey participant'spreferences as to items in the subset of items presented, determiningthat a particular item in the plurality of items is relatively neweritem than other items in the plurality of items, and adjusting thepredetermined selection algorithm so as to increase the likelihood thatthe relatively newer item will be selected during a subsequent selectionoperation for a subsequent survey. In this manner, a relatively neweritem will be presented more frequently so as to obtain additional surveydata for such newer item.

According to this and various other aspects of the invention, the methodmay further include a step of receiving new item information input by asurvey participant corresponding to the submission of a new item forinclusion in the plurality of items for the survey, such that the newitem is the relatively newer item. The relatively newer item may bedetermined according to the time of inclusion of the item in theplurality of items, compared with other items. The relatively newer itemmay also be determined according to the number of times that the itemhas been presented in prior surveys. The relatively newer item may alsobe determined according to the frequency that the item has beenpresented in prior surveys. It will be appreciated that the “frequency”an item is presented is not the same thing as the number of times anitem is presented, for example, a frequency of presentation could be“this item should be presented in 3 out of every 10 surveys,” while thenumber of times presented could merely be an absolute number.

According to a fifteenth aspect, the present invention is a method forconducting a computer-implemented survey relating to a plurality ofitems from a plurality of survey participants, with random selection ofitems biased by the need for more data. The invention of this aspectcomprises a number of steps that include storing information regardingthe plurality of items in a memory, the stored information includingdisplay information about each of the plurality of items forpresentation to a survey participant and frequency informationindicating a number of times that items in the plurality of items havebeen presented in a survey, biasing the plurality of items as a functionof the frequency information in anticipation of a selection operation,conducting a probabilistic selection operation comprising a randomselection within the plurality of items to select a subset of items forpresenting to a survey participant, presenting the display informationcorresponding to the selected subset of items to the survey participantvia a survey user interface, and receiving rating information input bythe survey participant via the survey user interface indicating thesurvey participant's preferences as to items in the subset of itemspresented. In this manner, a biased random selection of items to bepresented is conducted so as to avoid presenting only newer, lessfrequently presented items to later survey participants and to ensurepresentation of some early ideas to such later survey participants.

According to a sixteenth aspect, the present invention is a method forconducting a computer-implemented survey relating to a plurality ofitems from a plurality of survey participants so as to favor selectionof items with a higher mean rating. The invention of this aspectcomprises a number of steps that include storing information regardingthe plurality of items, the stored information including displayinformation about each of the plurality of items for presentation to asurvey participant, determining a mean rating of items previouslypresented to survey participants, biasing the plurality of items as afunction of the mean rating of items from the mean rating determiningstep in anticipation of a selection operation, conducting a selectionoperation involving selecting a subset of items from the plurality ofitems for presenting to a survey participant in accordance with apredetermined function that utilizes a probability of selection,presenting the display information corresponding to the selected subsetof items to a survey participant via a survey user interface, andreceiving rating information input by the survey participant via thesurvey user interface indicating the survey participant's preferences asto items in the subset of items presented. In this manner, a biasedrandom selection of items to be presented is conducted so as to favorselection of items with a higher mean rating.

In accordance with this aspect of the invention in particular, but mayalso be applicable to other aspects, the biasing step involvesapplication of a rank influence factor (rif) that may be adjusted toincrease or decrease the probability of an item being selected based onthe ranking of one item in relation to another item.

The computations and selection algorithms provided by the presentinvention are not dependent on the manner in which the data is presentedto the participant. The participant may be asked to express hispreference about the items presented using a variety of user interfaceconcepts including rating each item independently, arranging severalitems in order of preference, or allocating a fixed number of pointsamong the items presented.

These and other objects, features, and advantages of the presentinvention may be more clearly understood and appreciated from a reviewof the following detailed description and by reference to the appendeddrawings and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 presents graphical flow of the usage of the present invention.

FIG. 2 presents a flowchart of the usage and processes of a typicalsession of a participant's interaction with the present invention.

FIG. 3 depicts an exemplary user interface for the present invention asimplemented on a web page

FIG. 4 presents an overview of the process of selecting items,presenting them to participants, and obtaining and recording ratings forthe items presented in accordance with the present invention.

FIG. 5 depicts an exemplary database structure that may be used inaccordance with the present invention.

FIG. 6 presents an overview of the item selection process according tothe present invention.

FIG. 7 presents an overview of the process for calculating the selectionscore of each item.

FIG. 8 presents an overview of the process for computing a probabilityindex for each item in accordance with the present invention.

FIG. 9 depicts an exemplary two dimensional array that may be used inaccordance with the present invention to represent the resulting pool ofitems.

FIG. 10 presents an overview of the logic required to properly select aunique sample set to be presented to a participant.

FIG. 11 presents an overview of the process used to dynamically select asample based on prior interest.

FIG. 12 presents an overview of the logical flow used to allowparticipants to rate a variable number of items.

OBJECTS OF THE INVENTION

It is an object of the invention to use statistical methods for thepurpose of selecting which items should be presented to a surveyparticipant. Statistical analysis is performed on the already collectedresponses, if any, to determine which items will be presented to asurvey participant.

It is another object of the invention to use the standard error of themean statistic to determine the confidence level in the mean rating ofan item that was presented to subset of the entire population. Then, theconfidence level is used to determine which items would benefit fromadditional ratings by additional participants and should be selected tobe rated by future participants.

It is another object of the invention to provide a system that isoperative for selecting items to present to a participant dynamicallyrather than having the questions predetermined at the time the survey iscreated. This dynamic selection feature is not dependent on how previousquestions were answered in the same survey response. Rather, the itemsare selected for presentation based on how many participants havepreviously rated each item in the list of items, and how closely thoseparticipants agree on the ratings to each item.

It is a further object of the invention to provide a system in whichparticipants in a survey can contribute items to the list of items suchthat the contributed items are available for presentation to futureparticipants in the same survey.

It is a further object of the invention to use an adjustment factor tocause items in the database to be selected more or less often dependingon the number of times the items have already been selected andpresented to previous participants.

It is a further object of the invention to use an adjustment factor tocause items to be selected more or less often based on the rankingobtained from previous participants of a survey.

It is a further object of the invention to use a random selection ofitems to be presented to participants of a survey.

It is a further object of the invention to use a self-adjustingselection process based on previous responses during the same survey.The process is self-adjusting in that the subset of items to bepresented to a participant is selected based on all previous responses,and the probability of each item in the database being selected isautomatically adjusted as new items are collected and previous items arerated by survey participants.

It is a further object of the invention to provide a system that isoperative for making a continuous transition from random selection ofitems to intelligent selection of items. Items are randomly selectedwhen the survey process begins because no data is available for any ofthe items. As the process continues, the probability of each item beingselected is continuously adjusted to a more intelligent selection asitems are rated by participants and response data is collected. Thetransition from random to intelligent selection is a continuous process,rather than a process where at some point in time or maturity of thesurvey, the selection becomes more intelligent.

It is a further object of the invention to provide a system that isoperative for enabling survey participants to return to the survey andchange their original responses, submit new items, and rate new itemsthat have been collected since the time of their previous response tothe survey.

It is a further object of the invention to provide a system operativefor selecting items to present to survey participants based on criteriadefined by the administrator. The criteria may include attributes of theitems, such as category, originator, priority, purpose, or any otherattributes that may be tracked by an administrator for each itemcollected in maintained in the database.

It is a further object of the invention to provide a system operativefor enabling a participant of a survey who is willing to rate more thanthe sample set of items to view and rate additional items.

It is a further object of the invention to provide a system operativefor selecting items to present to a participant of a survey that areknown to be of interest to the participant. The determination of what isknown to be of interest can be made by the administrator's specificationof selection criteria for particular groups of respondents.Alternatively, items that have words similar to the words used by theparticipant may be selected for presentation to the participant.

It is yet a further object of the present invention to provide a systemoperative for enabling survey participants to add comments to items thatwill be viewed by other survey participants during the course of thesurvey.

DETAILED DESCRIPTION OF THE INVENTION

I. Introduction

The present invention is generally directed to a computer-implementedsystem for estimating the preference of a list of items as perceived bya population of participants without having each participant rate eachitem available. The system may be implemented using a variety ofcomputer technologies including, but not limited to, the internet, WorldWide Web, email, client-server, and distributed systems.

The system of the present invention encourages participation by limitingthe number of items that each participant will rate, thereby reducingthe amount of time needed to complete the survey. The system usesstatistics to select the items for which a low statistical confidencehas been reached relative to other items. The items that have a lowconfidence level are those that have not been sufficiently rated, orthat have been rated several times and have received inconsistentresponses.

The system of the present invention may be used for numerousapplications, including consumer research, employee evaluations, humanresources, information systems planning, and architectural planning.Further, although items are described herein, it should be understoodthat the system of the present invention may be used to rank productconcepts, people in an organization, or any other item that may bebeneficially ordered by participant preference.

II. System Overview

FIG. 1 presents graphical flow of the usage of the present invention. Aprocess manager 110, serves as the administrator and overseer of theprocesses and systems afforded by the present invention. The processmanager 110 interacts with a computer system 120 to create an electronicsurvey and sends electronic invitations 130 via email to the prospectiveparticipants of the survey.

Participants 140 interact with the computer system 120 and are allowedto input items 150 to the database of items 500 as specified in thepresent invention. The computer system 160 executing programs that areimplementing the algorithms and methods in accordance with the presentinvention will then select a subset of items to be presented to, andrated by the participant 170.

The responses 180 that are received electronically from the participantsare then collected by the computer system 190 executing the programsthat are implementing the algorithms and methods in accordance with thepresent invention. The responses 180 are processed by the computer 195to produce reports of ranked items. The responses 180 are also processedin accordance with the present invention to adjust the selectionprobabilities of the plurality of items so future participants arepresented with the items that are in most need of additional data toimprove the confidence in the accumulated mean rating.

FIG. 2 presents a general overview of the system of the presentinvention. First at 210, a participant is presented with an invitationto participate in a survey. If the participant consents to participatingin a survey, the system selects a sample of items from the database 220and presents the items to the participant 230. The participant rates theitems according to the participant's preference 240. The participant isalso able to rate items that the participant generates and inputs intothe survey 250. When the participant indicates that the rating iscomplete, the results of the survey, including the new items, are storedin a database 260.

FIG. 3 depicts an exemplary user interface for the present invention asimplemented on a web page. In this example, each participant ispresented with a graphical user interface which contains a section toallow the participant to submit a new item 310. A sampling of items 320from the plurality of items stored in the database are selected inaccordance with the present invention. The participant is allowed rate330 each item and submit the results so that they may be stored and usedto further adjust the sample selection so that future participants willget a set of ideas where more data is needed.

FIG. 4 presents an overview of the process of selecting items,presenting them to participants, and obtaining and recording ratings forthe items presented in accordance with the present invention. Aplurality of items 410 are stored in a storage array 450, either inmemory or in a database. This invention provides a selection process 415which is used to select a subset of the plurality of items 420 to bepresented to the participants 425. Each participant 425 and 435 areallowed to specify a rating for each item presented 430. The ratings 430are then stored in the storage array 450 where the selection process 415will then compute selection values as specified in the present inventionso that the items can be selected for presentation to futureparticipants. Some of the participants may also choose to input a newitem 440, which is then incorporated into the storage array 450 andincluded in the selection process for subsequent participants.

FIG. 5 depicts an exemplary database structure 500 that may be used inaccordance with the present invention. Each item or item is stored in anitem table 510, which contains basic information about the item. Eachitem is identified by a unique identifier 520, in this instance named“ItemId”. The ItemId is a reference number that is used as the primarykey for each item table 510. The “Value” field 530 contains the actualtext of the item itself. Each item also contains a status field 540, inthis instance named “ItemStatusID”, which may optionally be stored in aseparate table 550. The status 540 is used as part of the criteria fordetermining if an item is eligible for selection, as will be describedin detail below.

The “TimesShown” field 560 is used in the computation of the confidencevalue for each item. The confidence value relates to the number of timesan item has been presented to participants, and will be discussed ingreater detail below.

When a participant inputs an item, the participant information may bestored in a different table, such as a “Users” table 570. This enablesefficient storage of participant information, particularly where a givenparticipant submits more than one item. Each item has a “UserID” field580 that uniquely identifies a participant record in the users table.

During the process of executing this invention, each participant ispresented a subset of the plurality of items to rate. The ratingssubmitted by each participant are stored in a response rating field 595.One row is added to the response table 590 for each item that isselected for a participant during the execution of this invention. Eachrow in the response table is initialized with a rating of zero. Once theparticipant actually rates an item, then the response table rating field595 is updated with the actual value for that item as set by theparticipant.

The response table 590 is also used to determine if the participant isreturning to survey as depicted in FIG. 6, box 645; in which case thesurvey participant will be presented with the same set of items thatwere previously selected for the participant.

It should be noted that the system of the present invention does notdepend on the mechanism by which the items are placed in the database.Thus, various database designs and storage mechanisms may be used asdesired. Such designs and mechanism are well known by those skilled inthe art and are not described herein.

An overview of the item selection process 600 is presented in FIG. 6.The system first determines whether the participant is visiting the sitefor the first time 605 by performing a query to the database to see if aresponse has already been received for the participant (by userid). Ifthe participant is visiting the site for the first time, a set of itemsis selected 1000 and presented to the user 615. An item is selected forpresentation to a participant if a statistical analysis of the dataassociated with the item indicates that more data is needed to improvethe statistical level of certainty, or “confidence level” relative tothe other items in the database. The confidence level will be increasedfor an item when more participants have rated an item or when theparticipants who have rated the item increasingly agree on the level ofdesirability or rating of the item. As such, there is no particularthreshold or other absolute value to determine when enough participantshave rated an item or when the participants who have rated an item agreeenough.

Instead, a confidence score is computed for each item, as will bediscussed in connection with FIGS. 6-10. The confidence scores are usedonly to compare the level of confidence among two or more items todetermine which items are most needing additional data. The items thathave the lower confidence scores are the items that could benefit themost from additional ratings by participants. For example, if an itemhas been rated by many participants and the ratings mostly agree, theitem would have a higher confidence score than an item that has onlybeen rated by a few participants or that has been rated vastlydifferently.

After the items are selected 1000 and presented 615, the participant mayexpress his or her preferences for the various items presented 620. Theselection process 1000 is described in detail in Para 108 and FIG. 10.Additionally, the participant may be afforded the opportunity to inputone or more additional items, which are also rated by the participant.The responses are then stored 625 and entered into a response table 630.Participants are allowed to input new items as shown in 670 and 680.FIG. 6 illustrates a participant's ability to input a new item afterrating existing items. The inputting of new items could also occurbefore the participant has rated items. In either case, each item inputby a participant is stored in the item table 640 where it is availableto be presented to another participant. Thus, an item input by oneparticipant may be rated by one or many other participants.

The participants who are rating items may optionally return to thesystem at any time during the course of the survey program andparticipate again. When a participant returns to the system for a secondor subsequent visit, the items previously evaluated by the participantare retrieved 635 from the item table 640. Likewise, the participant'sprevious responses are retrieved 645 from the response table 650 usingthe UserId created. The previous items and responses are then displayed655. The participant is then able to modify the responses if desired660. The responses are then stored 625 and entered in the response table630. The new data is used in all subsequent computations of confidencefactors.

III. Determining the Selection Score for Each Item

The selection score for an item determines the desired probability thatit will be selected by future participants. There are severalintermediate computations need to arrive at the selection score. Thissection describes how the response data from participants who havealready rated an item are used to compute the intermediate values andultimately the selection score.

First a mean of the responses is computed, and then an estimate of thestandard deviation of the mean is computed. Once the standard deviationof the mean value is available, the standard error of the mean can becomputed. The standard error of the mean is then translated to aconfidence factor which is a representation of the amount of confidencewe have in the accuracy of the previously accumulated responses on ascale of one to one hundred.

Before the confidence factor is used to compute a selection score, twoadditional factors are used to give the administrator of the systemadditional control over which items should be preferred for selection.The participation influence factor is an arbitrary number, specified bythe administrator, which controls the amount of weight to be given tothe count of the number of people who responded. The rank influencefactor is an arbitrary number, specified by the administrator, whichallows higher ranked items to have greater preference in the selectionprocess.

A. Calculation of the Confidence Factor

FIG. 7 presents an overview 700 of the process for calculating theselection score of each item. As stated above, the selection score isused to determine whether a particular item will be selected forpresentation to a participant. First, an arithmetic mean (μ) iscalculated for the responses already collected for the item 710. Thearithmetic mean of the rating values is referred to as the “mean ratingvalue”. The mean rating value provides a consolidated rating for allparticipants who rated the element. A mean rating value is notcalculated until there are at least two responses ranking the item. Themean rating is calculated as follows:

$\mu = \frac{X_{1} + X_{2} + X_{3} + \ldots + X_{n}}{n}$

where X is the rating value and n is the number of times the item hasbeen rated.

As data is collected over time, a “rolling” mean rating (μ′) iscalculated using the previous mean rating (μ′), the new rating value(Xn), and the number of times (n) this item has been rated as follows:

$\mu^{\prime} = \frac{{\mu\left( {n - 1} \right)} + X_{n}}{n}$

Next at 720, the statistical standard deviation (s) of the mean ratingis calculated. The standard deviation of the mean rating represents theamount of agreement or disagreement among the population of participantswho rated the item. The standard deviation is calculated using the“nonbiased” or “n−1” method. Although only a subset of the entirepopulation of participants actually rated the item, this methodestimates the standard deviation for the entire population.

$s = \sqrt{\frac{{n{\sum x^{2}}} - \left( {\sum x} \right)^{2}}{n\left( {n - 1} \right)}}$

Next, the standard error of the mean response is computed 730. Thestandard error is calculated from the estimated standard deviation ofthe mean calculated above and the number of responses included in thecomputation of the mean as follows.

$\sigma_{M} = \frac{s}{\sqrt{n}}$

where s is the estimated standard deviation of the mean rating and n isthe number of times the item has been presented for evaluation. As canbe readily seen from the equation set forth above, the number ofparticipants responding to each item, and the degree to which they agreeor disagree determines confidence level in the data for a given item. Asthe standard deviation decreases, the standard error also decreases.

The standard error will have a value between 0 and the maximum ratingvalue of the item, inclusive. The maximum rating value may vary for eachsurvey application as desired. Thus, for example, if the item can begiven a rating from 1 to 5, the maximum standard error will be 5. Theitems with the highest standard error are preferred in the selectionprocess because more data is needed to increase the level of certaintyin the preference ranking. As the number of data points for an itemincreases, the standard error of the data collected for the itemdecreases. As the standard error of the data for an item decreases, theconfidence in the mean rating increases and the data collected moreaccurately estimates the statistical parameter of the population.

Next, a confidence factor (cf) is calculated for the item 740. Thesystem always computes a confidence factor for each item regardless ofhow many items are stored in the item database at the time of thecomputation. The confidence factor is used to measure the need to obtainadditional data for each item relative to the need to collect additionaldata for all other items. When items are selected to be presented, theitems with the lowest confidence levels are preferred for selection.

The confidence factor is calculated using the standard error of the meanresponse and the maximum possible rating value for each item as follows:

${cf} = {100 - \frac{100\sigma_{M}}{M}}$

where M is the maximum possible rating value of the item. The confidencefactor (cf) has a value between 0 and 100. A confidence factor of 100indicates maximum confidence in the mean rating value for an element.Theoretically, this is only achievable if an item is rated by eachparticipant in the population, and each participant provides the samerating for the item.

A confidence factor is calculated for an item when as few as tworesponses are collected for the item. For items with less than tworesponses, the confidence factor is set to 0, which causes them to befavored for selection over the items that have at least two responses.Once an item has received two responses, the probability of selection iscomputed relative to all other items in the database. As additionalitems are added, the confidence factors are recomputed for all items.

According to the present invention, items that are added to the databaseduring the survey process are given the opportunity to achieve fairrankings quickly with minimal bias or skewing of the data. A lowconfidence factor will be computed for later arriving items because theyhave fewer respondents, thereby causing the newer items to be selectedmore frequently than earlier arriving items. The newer items will beselected more frequently until the confidence in the data collected forthe newer items gains equality with the earlier items. As such, thesystem of the present invention provides a significant advantage overtraditional survey methods in which new items are always at adisadvantage over items that were in the database from the beginning ofthe process. In such traditional systems, a set of items must becompiled before the process begins, and if new items are collected afterthe process begins, the new items must wait for a second survey.

B. Use of the Participation Influence Factor to Influence the ConfidenceScore

Still viewing FIG. 7, according to another aspect of the presentinvention, the system administrator may specify an additional factor 750that will cause the number of participants rating an item to have moreinfluence in the confidence score than would otherwise be computed usingthe confidence factor alone. This additional factor is called the“participation factor” (pf).

The participation factor is calculated using the number of times theitem was presented and the total number of times all items werepresented as follows:

${pf} = {\frac{N}{P} \times {cf}}$

where N is the number of times the item was presented, and P is thetotal number of presentations of all items.

A “participation influence factor” (pif) may be used to control thedegree to which the participation factor influences the confidencescore. The participation influence factor can be adjusted to give moreor less weight to the number of participants who responded. When theparticipation influence factor is adjusted to a high value, the numberof people who have provided preference data for an item becomes thedominant factor in the computation of confidence in the data collectedfor that item. Also, when the participation influence factor is adjustedto a high value, the standard deviation, or amount of agreement amongthe people who have provided preference data, becomes less of a factorin the computation of the confidence in the collected data for the item.The participation influence factor can be adjusted to a neutralposition, which causes the confidence factor to be computed using onlythe generally accepted calculation for standard error.

To control the degree to which the participation factor influences theconfidence score, an additional factor is introduced. This “adjustedparticipation factor” (apf) is calculated as follows.

${apf} = {\frac{N}{P} \times {cf} \times {pif}}$

As is readily observed, if the participation influence factor is set to0, the adjusted participation factor will be 0 and have no influence onthe confidence score.

C. Calculation of the Confidence Score

Next, the confidence score (cs) for a particular item is computed 760.The confidence score is a measure of the relative amount of confidencein the statistical mean rating calculated from the data provided by theparticipants who rated the element. It should be noted that theconfidence score cannot be computed until at least two participants haverated an element. The confidence score is calculated for each element asfollows:Cs=cf+apf

The system of the present invention does not use the confidence score todetermine an absolute selection order. Rather, it uses the confidencescore adjust the probability that each item will be selected. This willcause some items that already have a higher confidence factor in thedata collected to be selected and presented to participants along withthe newer items with lower confidence in the data. Without thisprobabilistic approach, it would be likely that newer items would beselected and presented only to newer participants and existing itemswould only be selected for rating by early participants. Thus, thepresent invention enables a more random selection of items to bepresented, while showing items that need additional data morefrequently.

D. Use of Adjusted Mean Rating and the Rank Influence Factor toInfluence the Confidence Score

In some instances, it might be desirable to favor the selection of itemswith a higher mean rating over items with a lower mean rating. Forinstance, there may be a situation in which there is a greater need forcertainty about the order of preference of high ranking items, and thereis little or no concern about the order of preference of low rankingitems. By way of specific example, a survey may be initiated to identifyitems in which to invest resources in. In such an example, there wouldbe little interest in low ranking items because such items will not beconsidered. However, if the objective is to rank employees for thepurpose of terminating the lower ranks, the need for confidence in thelower rankings is equally as important as the higher rankings.

In either of such instances, the system still uses the confidence scorecomputed above to determine which elements need more data. However, theuse of the adjusted mean rating (amr) and rank influence factor (rif)enable the accumulated mean rating of an element to have a controlledamount of influence on its selection score.

First, according to one aspect of the present invention, the ratings fora set of items may be rescaled between the minimum and maximum ratingsto more clearly discern the order of preference between the items 770.The rescaled mean rating is called the “adjusted mean rating”. Theadjusted mean rating (amr) is computed as follows:

${amr} = \frac{100 \times \left( {R_{\max} - R_{\min}} \right)}{\mu}$

where R_(max) is the maximum rating that was given to the items in therepository, and R_(min) is the minimum rating that was given to theitems in the repository.

Second, according to yet another aspect of the present invention, theamount of influence that the adjusted mean rating has on the selectionscore may be controlled by applying an externally controlled factorcalled the “rank influence factor” 780. The rank influence factor (rif)may be adjusted to increase or decrease the probability of an item beingselected to be presented to a participant based on the actual ranking ofthe item in relation to the other items. If the rank influence factor isset to a high value, items with a higher current ranking are more likelyto be selected. If the rank influence factor is set to 0, the adjustedmean rating will have no influence on the selection score, as will bedescribed in detail below.

E. Calculation of the Selection Score

Finally, the selection score (SS) is calculated 790 as follows todetermine which items should be preferred in the selection process:

${SS} = {{cs} - \frac{{rif} \times {amr} \times {cs}}{10000}}$

The selection score is calculated for each item in the item database.The items that will be selected are those with the lowest scores, aswill be described in further detail below.

IV. The Item Selection Process

A. Overview

After a selection score is computed for each item, the desiredprobability that a given item will be presented is computed.Specifically, the selection score calculated above determines theprobability that an item will be selected and, therefore, the frequencyat which it will be presented to participants.

For purposes of explanation only, the selection process may be comparedto placing numbered balls in a barrel and randomly drawing balls out ofthe barrel. For instance, if 100 balls, each having a unique numberbetween 1 and 100 are placed in the barrel, each has a 1% chance ofbeing drawn from the barrel. To increase the probability of a numberbeing selected, more balls with the same number are placed into thebarrel. Instead, if there were 100 balls in a barrel and 30 of them arenumbered “12”, a random selection from the barrel would effect a 30%chance of drawing a ball with the number 12 on it.

According to one aspect of the present invention, no assumptions aremade about the number of responses that will be ultimately received fora given item. Instead, the system of the present invention selects thebest sample set based on information available at the time of selection.Likewise, the total number of responses that will be collected for theset of items or for any particular item is unknown at any time duringthe process. Thus, the system of the present invention uses the numberof responses already collected when computing the confidence factor andselection probabilities. Sample selections are made based onprobabilities that were computed just prior to the selection and theitems that are in most need of additional data at that time are the mostlikely to be selected for the sample.

B. Calculation of the Probability Index

FIG. 8 presents an overview of the process for computing a probabilityindex for each item in accordance with the present invention.

First, using the selection score calculated above 810, a normalizedscore (Sn) is computed 820 for each item as follows:

${Sn} = {100\left( {1 - \frac{SS}{\sum{ss}}} \right)}$

The normalized score has a value between 1 and 100. As the valueapproaches 100, the probability of selection of the item increases.

If the sum of the selection scores is zero, then the normalized score isset to 100. The sum of the selection scores will be zero when none orthe items have been rated, or when all ratings are zero. In eitherinstance, the normalized score for each and every item will be set at avalue of 100, thereby providing each item an equal opportunity to beselected for presentation to a participant.

In many instances, the normalized score is concentrated around arelatively small number of scores, e.g. between 90 and 100. In thisinstance, a distributed score (Sd) may be calculated 830 for each itemacross a range of values from 1 to 100 as follows:

${Sd} = {100\frac{{Sn} - {Sn}_{\min} + 1}{{Sn}_{\max} - {Sn}_{\min}}}$

It should be noted that if Sn max is equal to Sn min then all items havethe same rating. This could occur when none of the items have been ratedor when all items have the same mean rating. In either case, thedistributed score for each and every item is set to 100, thereby makingeach item equally available for selection.

The desired probability of selection (S_(prob)) is then computed 840 foreach item as follows:

$S_{prob} = \frac{Sd}{\sum{Sd}}$

The probability of selection of each item is then multiplied by the sizeof the item selection pool, as defined by the administrator or managerof the system, to determine how many times the item should berepresented in the set. This value, called the “probability index” (Si),is calculated 850 as follows:Si=S _(prob)×PoolSize

A selection pool size of at least 1000 is recommended to avoid excessiverounding error that could skew the results. However, the actual poolsize may be increased if the number of items is expected to be greaterthan 1000. After the pool is created and each item is represented in thepool the number times as indicated by its probability index value, arandom selection of items from the pool will yield the desired results.

C. Representation of the Pool of Items

Turning to FIG. 9, a two dimensional array 900 may be used in accordancewith the present invention to represent the resulting pool of items. Thearray 900 contains one row for each item. Each item in the tableincludes a low index number 910 and a high index number 920. These indexnumbers represent how many times each item is represented in the pool.For example if an item had a low index value of 10 and a high indexvalue of 15, the item would be represented six times in the pool. Thetable could be compressed even further by only storing the high index.By storing the low index and the high index for each item the desiredselections can be efficiently processed by obtaining a random numberthen using the following SQL Select statement: SELECT TOP 1 ITEM FROMPOOLARRAY WHERE RANDOM_NUMBER BETWEEN LOW_INDEX and HIGH_INDEX. Thisstatement will select the single item where the random number fallsbetween the low and high indexes for that item.

The process of using a random number to select an item will be repeatedenough times to retrieve the desired number of items according to thesample size that is requested. To fill a sample set with the desirednumber of unique items that all meet predetermined criteria, the logicis somewhat more sophisticated than a simple loop that repeats a fixednumber of iterations.

D. Selection of the Sample Set

FIG. 10 presents an overview 1000 of the logic required to properlyselect a unique sample set having the desired sample size, and in whicheach item meets the specified selection criteria according the status ofthe items.

First, the desired sample size is obtained 1005. This is the maximumnumber of items that will be selected and presented to a participant.The number of items will be equal to the sample size unless the numberof items that are available and eligible for selection is less than thesample size. The sample size is a parameter that is specified by theadministrator of the system.

Next, the system determines whether the number of items in the databaseis less than or equal to the sample size 1010. If the number of items inthe database is less than the number of items in the sample size, allitems that meet the selection criteria are inserted into the sample set1015.

The selection criteria are specified by the administrator and consist oflogical conditions based on the attributes of each item. For example,only items in an “active” status may be eligible for selection. Items ina “declined” status would not meet the selection criteria. Variousconditions may be specified by the system administrator as desired. Ifthe status or attributes of an item change during the survey processsuch that the item meets the defined criteria, the item becomes eligiblefor selection. If the criteria for selection changes during the process,all items that meet the criteria then become eligible for selection.When items become eligible for selection, a confidence factor iscomputed for the items, and the items are selected as described above.

If the number of items in the database is not less than the number ofitems in the sample size, the counter is set to a value of 0 and theselection process continues 1020.

Next, if the counter is less than the sample size 1025, a random numberbetween 0 and the maximum probability index value is then generated1030.

Next, using the random number generated in step 1030, an item isselected from the pool 1035. The item selected has a low index valueless than or equal to the random number and a high index value greaterthan or equal to the random number. If the selection pool table wasconstructed properly, such as that in FIG. 9, one and only one item willqualify for selection.

Next, still viewing FIG. 10, the item selected above is compared to anyitems already selected for the sample set 1040. If the item waspreviously selected for the sample set, the system returns to step 1030and repeats the process until an item is selected that has not alreadybeen selected for this sample set.

Next, the system verifies that the selected item meets any selectioncriteria specified by the administrator 1045. If the item does not meetthe selection criteria, the item will not be made part of the sampleset. The process then repeats until items are selected that meet theselection criteria.

Next, the selected item is inserted into the set of items for the sampleset 1050. Additionally, a value of 1 is added to the counter for thenumber of items in the sample set 1055. If the number of items now inthe sample set is equal to the desired sample size, the selection of thesample set is complete and the items are displayed to the participant1060. If not, the process repeats until the desired sample size isattained.

E. Selecting Items within a Participant's Scope of Interest

According to another aspect of the present invention, a set of items maybe presented to a participant based on the particular interests of theparticipant. The various interests of a participant can be determined bythe system either statically or dynamically.

To determine a participant's interests statically, the participant ispresented with a query about the participant's interests. Then,selection criteria specified by the administrator restrict the number ofitems that are eligible for presentation to a given participant. Thesecriteria are used during the selection process previously described inconnection with in FIG. 10.

Dynamic determination of a participant's interest is conducted byexamining the contents of items previously submitted by the sameparticipant, and selecting other items relating to the subject mattersubmitted. For example, if a participant submits several items relatedto the topic of “security”, the system will select other items relatedto security for presentation to the participant.

FIG. 11 illustrates the process 1100 used to dynamically select a samplebased on prior interest. First, an empty pool of items that are eligiblefor selection for the participant is created 1110. The pool is thenpopulated with appropriate items, and used as the selection pool in theselection process as previously described in connection with FIG. 10.

After the participant has input the rating information, a query is madeto the database of items to determine keywords relating to items thatwere submitted by the current participant 1120. This query can beadjusted by the administrator to also include items that have been ratedby the current participant where the rating value for those itemssurpasses a specified level. When a participant gives an item a highrating, this can be used as an indication that the participant hasinterest in the type of item or subject of the item and can therefore beconsidered a good participant to rate other items of the same or similartype or subject matter.

Each of the items that match the query is used in a similar manner 1130to find other items in the database 1140 that meet a specified degree ofsimilarity. The results of this search are then used to populate theselection pool 1150, which is then used as the selection pool for theprocess depicted in FIG. 10.

VI. Allowing Participants to Rate a Variable Number of Items

According to yet another aspect of the present invention, a participantis able to view and rate more items than provided in the subsetpresented. This provides significant advantages over presently availablesurvey systems, which present too many questions to the participant andrisk losing the participant. According to the present invention, asmaller, more reasonably sized, subset of items may be selected for aninitial presentation to the participant, who can then choose to view andrate additional items if desired.

FIG. 12 presents an overview 1200 of the logical flow used to allowparticipants to rate a variable number of items.

First, a sample size for the participant is determined 1210. If desired,the system administrator may allow the participant to choose how manyitems to view or rate before the process begins. In this case, theparticipant is simply selecting the sample size. After the sample sizeis selected 1220, the process continues with items being selected andpresented 1230 as described above.

Additionally, the participant may view and rate a set of items, and thenchoose to view and rate additional items. In this instance, after theparticipant has rated the first set of items presented, the participantis offered an opportunity to request another sample 1240. The samplesize may or may not be fixed, and may be established by theadministrator or participant as desired. If the sample size is fixed,the administrator may specify the initial sample size, and the size ofany subsequent samples selected for a participant. Likewise, if thesample size is selected by the participant, the administrator mayspecify the upper and lower limits of the sample size.

It will be understood that the foregoing relates only to the preferredembodiments of the present invention, and that numerous changes may bemade therein without departing from the spirit and scope of theinvention as defined by the appended claims.

What is claimed is:
 1. A method for conducting a computer-implementedsurvey to collect survey information relating to a plurality of itemsfrom a survey participant by the display of information relating to aselected subset of a set of items in the survey to a survey participantvia a survey user interface generated by a computer, the selected subsetof items selected as a function of the number of times that particularitems have been previously shown to other survey participants in one ormore prior surveys, comprising the computer-implemented steps of:storing information regarding the set of items in a memory, the storedinformation including (a) display information about each of theplurality of items in the set of items for presentation to a surveyparticipant and (b) presentation number information corresponding to thenumber of times that each particular item in the set of items has beenpreviously shown to survey participants in one or more prior surveys;selecting a subset of items from the set of items for presenting to asurvey participant in accordance with a predetermined selectionalgorithm that utilizes the presentation number information to selectitems that are in need of additional data to improve a confidence factorrelative to other items in the set of items; presenting the displayinformation corresponding to the selected subset of items to the surveyparticipant via the survey user interface; and receiving ratinginformation input by the survey participant via the survey userinterface indicating the survey participant's preferences as to items inthe selected subset of items presented.
 2. The method of claim 1,further comprising the step of utilizing the rating information input bythe survey participant to affect the probability of selection of theitems in the selected subset in a subsequent selection of a subset ofitems for presentation to a subsequent survey participant using thepredetermined selection algorithm.
 3. The method of claim 1, wherein thepredetermined selection algorithm utilizes an adjustment factor to causeitems to be selected more or less often as a function of the ratinginformation obtained by previous participants of a survey.
 4. The methodof claim 1, wherein the predetermined selection algorithm utilizes, atleast in part, a random number selection of items in the plurality ofitems.
 5. The method of claim 4, wherein the plurality of items includesa predetermined duplication of items in a set of the plurality of items,with the number of duplications of particular items influenced by therating information.
 6. The method of claim 1, wherein the predeterminedselection algorithm is self-adjusting based on previous responsesreceived during a previous survey of the plurality of items.
 7. Themethod of claim 6, wherein the self-adjusting is based on modifying theprobability of selecting an item from the plurality of items, themodifying in turn based collection of new items input by surveyparticipants and rating of new items in comparison of previouslyincluded items.
 8. The method of claim 1, wherein the predeterminedselection algorithm is operative initially to randomly select items forthe subset of items, and thereafter operative to select items based onutilization of the rating information.
 9. The method of claim 8, whereinthe probability of selection of a given item is continuously adjusted asitems are rated by survey participants.
 10. A method for conducting acomputer-implemented survey to collect survey information relating to aplurality of items from a survey participant by the display ofinformation relating to a selected subset of a set of items in thesurvey to a survey participant via a survey user interface generated bya computer, the selected subset of items selected as a function of thenumber of times that particular items have been previously shown toother survey participants in one or more prior surveys, comprising thecomputer-implemented steps of: storing information regarding the set ofitems in a memory, the stored information including (a) displayinformation about each of the plurality of items for presentation to asurvey participant and (b) frequency of display informationcorresponding to the number of times that each particular item in theset of items has been previously shown to survey participants in one ormore prior surveys; selecting a subset of items from the set of itemsfor presenting to a survey participant in accordance with apredetermined selection algorithm that utilizes the frequencyinformation to select items that are in need of additional data toimprove a confidence factor relative to other items in the set of items;presenting the display information corresponding to the selected subsetof items to the survey participant via the survey user interface;receiving rating information input by the survey participant via thesurvey user interface indicating the survey participant's preferences asto items in the selected subset of items presented; and utilizing therating information input by the survey participant to affect theprobability of selection of items in the set of items for a subsequentselection of a subset of items for presentation to a subsequent surveyparticipant using the predetermined selection algorithm.
 11. The methodof claim 10, wherein the rating information is selected from the groupcomprising: ranking of items relative to each other, ranking of theitems on a scale, grading the items, ordering of the items, allocatingpoints among items, scaling the items, choosing an item over otheritems, categorizing items, and other equivalent methods of indicating apreference of one item over another.
 12. The method of claim 10, whereinthe step of selecting a subset of items in accordance with thepredetermined selection algorithm comprises selecting based on a rankingof items using rating information from previous participants, such thatthe probability of selection of particular items for presenting in asubsequent survey is influenced by the rating information, wherebycertain items that have been rated lower than other items are morelikely to be selected for a survey so as to increase the number ofpresentations of such items.
 13. The method of claim 10, furthercomprising the step of providing the rating information for each item asan output of the method indicative of survey results.
 14. The method ofclaim 10, wherein the memory is a random access memory array.
 15. Themethod of claim 10, wherein information regarding the plurality of itemsis stored in an ordered array and selected according to a probabilityindex.
 16. The method of claim 15, wherein the information about eachitem in the ordered array is stored in a data field in the orderedarray.
 17. The method of claim 10, wherein the subset of items selectedfor presentation to the survey participant is an initial subset, andwherein the subsequent selection of a set of items for presentationcomprises a selection from the plurality of items that may include oneor more of the items from the initial subset.
 18. The method of claim10, wherein the information regarding the plurality of items includes aunique identifier for each item for use as a primary key to access theitem in the memory.
 19. The method of claim 10, further comprising thestep of storing a users item table for storing information provided by asurvey participant relating to an additional item for inclusion in theplurality of items.
 20. The method of claim 10, wherein the informationregarding the plurality of items includes status information about eachitem.
 21. The method of claim 20, wherein the status information isindicative whether an item has been previously shown to a surveyparticipant or not.
 22. The method of claim 21, wherein thepredetermined selection algorithm also utilizes the status informationin conjunction with the frequency information.
 23. The method of claim10, wherein the frequency information is stored in a times-shown fieldfor each item in the array.
 24. The method of claim 10, wherein thepredetermined selection algorithm utilizes a selection score inselecting items for presentation.
 25. The method of claim 24, whereinthe selection score is based upon a confidence score.
 26. The method ofclaim 25, wherein the selection score is further based on an adjustedmean rating determined from the rating information.
 27. The method ofclaim 25, wherein the selection score is further based on a rankinfluence factor.
 28. The method of claim 27, wherein the rank influencefactor is an arbitrary number used to adjust the probability of an itembeing selected based on ranking information.
 29. The method of claim 28,wherein the ranking information comprises information corresponding tothe actual ranking of an item in relation to other items in theplurality of items.
 30. The method of claim 10, wherein the selectionalgorithm selects an item from the plurality of items based on acomputation of a probability index.
 31. The method of claim 30, whereinthe probability index is determined based on a normalized selectionscore.
 32. The method of claim 31, wherein the normalized selectionscore is utilized to determine a probability of selection for each itemin the plurality of items, the probability of selection of each item isused to determine how many times an item is represented in the pluralityof items for selection.
 33. The method of claim 10, wherein theplurality of items are represented in a computer system as a pool ofselectable items stored in an array of items, with each item in the poolhaving a high index number and a low index number, with the indexnumbers representing how many times an item is represented in the poolof selectable items, and wherein the step of selecting a subset of itemscomprises selecting from the pool of items based on a random number usedto index into the array of index numbers.
 34. The method of claim 33,wherein the subset of items for presenting to a survey participant isselected by repeating the step of selecting utilizing the random number,until a predetermined number of items corresponding to the size of theselected subset of items has been chosen for presentation.
 35. Themethod of claim 10, wherein a selected subset of items comprises aunique sample of items in the plurality of items of a predeterminedsample size that meets predetermined selection criteria according tostatus information associated with the item.
 36. The method of claim 35,wherein the predetermined sample size comprises the maximum number ofitems presented to a survey participant in the survey.
 37. The method ofclaim 35, wherein the status information comprises informationindicative of a condition associated with an item.
 38. The method ofclaim 10, wherein the status information is selected from the groupcomprising: whether an item is scheduled, whether an item has beenapproved, whether an item is implemented, whether an item is active,whether an item is in or under review, whether an item has beensubmitted, whether an item has been declined, or other equivalentinformation indicative of a condition of an item.
 39. The method ofclaim 10, wherein the selected subset of items is selected forpresentation based at least in part on an indication of interest of aparticipant.
 40. The method of claim 39, wherein the indication ofinterest of a participant is obtained by input of interest informationby a survey participant in response to a query prior to selection of thesubset.
 41. The method of claim 39, wherein the indication of interestof a participant is obtained by examining items previously submitted bythe survey participant, and by selecting other items from the pluralityof items based on the topical similarity of other items in the pluralityof items.
 42. The method of claim 39, wherein the indication of interestof a participant is obtained by executing a query of keywords relatingto items submitted by the survey participant.
 43. The method of claim10, wherein the selected subset of items for presentation is a firstselected subset of a predetermined small number of items, where “small”is relative but determined based on a number that is deemed by a surveymanager to be acceptable for purposes of a particular survey, andfurther comprising the step of selecting additional items forpresentation to a survey participant.
 44. The method of claim 43,wherein the step of selecting additional items for presentation to asurvey participant is based on information provided by a surveyparticipant indicating a desire to view and rate more items.
 45. Themethod of claim 44, wherein the survey participant is provided with adisplay offering an opportunity to request an additional sample of itemsfor rating, and wherein the information provided by the surveyparticipant indicating a desire to view and rate more items is input bythe survey user interface.
 46. The method of claim 45, wherein theopportunity to request an additional sample of items for rating isproviding during a survey session.
 47. A method for conducting acomputer-implemented survey to collect survey information relating to aplurality of items from a survey participant by the display ofinformation relating to a selected subset of a set items in the surveyto a survey participant via a survey user interface generated by acomputer, the selected subset of items selected as a function of thenumber of times that particular items have been previously shown toother survey participants in one or more prior surveys, comprising thecomputer-implemented steps of: arranging information relating to the setof items in a memory as entries in an ordered array; providing a uniqueidentifier for each entry in the array; providing display informationabout each item in the array entry for presentation to a surveyparticipant; providing a status information field for each item in thearray; providing a times-shown field for each item in the array;selecting a subset of items from the set of items for presenting to asurvey participant in accordance with a predetermined selectionalgorithm that utilizes the information in the times-shown field of theitems in the set of items to select items that are in need of additionaldata to improve a confidence factor relative to other items in the setof items; presenting the display information corresponding to theselected subset of items to the survey participant via the survey userinterface; receiving rating information input by the survey participantvia the survey user interface indicating the survey participant'spreferences as to items in the subset of items presented; and utilizingthe rating information input by the survey participant to affect theprobability of selection of the items in the selected subset of itemsfor a subsequent selection of a subset of items for presentation to asubsequent survey participant using the predetermined selectionalgorithm.
 48. A method for conducting a computer-implemented survey tocollect survey information relating to a set of items from a pluralityof survey participants by the display of information relating to itemsin the survey to a survey participant via a survey user interfacegenerated by a computer, comprising the computer-implemented steps of:storing information regarding each item in the set of items in a memory,the stored information including display information about each item inthe set of items for presentation to a survey participant via the surveyuser interface; conducting a selection operation involving selecting afirst subset of items for presenting to a survey participant inaccordance with a function that utilizes a probability of selection;presenting the display information corresponding to the selected firstsubset of items to the survey participant via the survey user interface;receiving rating information input by the survey participant via thesurvey user interface indicating the survey participant's preferences asto items in the first subset of items presented; determining anadjustment factor for the probability of selection of items for a subsetof items as a function of the number of times that the items havepreviously been selected and presented to previous survey participants;and applying the adjustment factor to the probability of selection for asubsequent selection operation for selecting a second subset of itemsfor a subsequent survey.
 49. A method for conducting acomputer-implemented survey to collect survey information relating to aset of items from a survey participant by the display of informationrelating to a selected subset of the set of items in the survey to asurvey participant via a survey user interface generated by a computer,comprising the computer-implemented steps of: storing informationregarding each item in the set of items in a memory, the storedinformation including display information about each items in the set ofitems for presentation to a survey participant via the survey userinterface and frequency information indicating a number of times thateach item in the set of items has been presented in a survey; biasingthe set of items as a function of the frequency information inanticipation of a selection operation; conducting a probabilisticselection operation comprising a random selection within the set ofitems to select a first subset of items for presenting to a surveyparticipant; presenting the display information corresponding to theselected first subset of items to the survey participant via the surveyuser interface; and receiving rating information input by the surveyparticipant via the survey user interface indicating the surveyparticipant's preferences as to items in the first subset of itemspresented, whereby a biased random selection of items to be presented tothe survey participant is conducted so as to avoid presenting onlynewer, less frequently presented items to later survey participants andto ensure presentation of some earlier items to such later surveyparticipants.